AIAA-2005-1 130 


Development of a Work Control System for Propulsion Testing at 

NASA Stennis 

Elizabeth A. Messer 1 " 

NASA, John C. Stennis Space Center, Mississippi, USA 


In 1996 Stennis Space Center was given management authority for all Propulsion Testing 
for NASA. Over the next few years several research and development (R&D) test facilities 
were completed and brought up to full operation in what is known as the E-Complex Test 
Facility at Stennis Space Center. To construct, activate and operate these test facilities, a 
manual paper-based work control system was created. After utilizing this paper-based 
work control system for approximately three years, it became apparent that the research 
and development test area needed a better method to execute, monitor, and report on tasks 
required to further propulsion testing. The paper based system did not provide the 
engineers adequate visibility into work tasks or the tracking of testing or hardware 
discrepancies. This system also restricted the engineer’s ability to utilize and access past 
knowledge and experiences given the severe schedule limitations for most R&D propulsion 
testing projects. Therefore a system was developed to meet the growing need of Test 
Operations called the Propulsion Test Directorate (PTD) Work Control System. This system 
is used to plan, perform, and track tasks that support testing and also to capture lessons 
learned while doing so. 


TPS 

= Test Preparation Sheet 

DR 

= Discrepancy Report 

DOP 

= Detailed Operating Procedure 

PTD 

= Propulsion Test Directorate 

SSC 

= Stennis Space Center 

CR 

= Change Request 

TR 

= Test Request 

WCS 

= Work Control System 


Nomenclature 


I. Introduction 

T HIS paper will explain the requirements and steps taken to develop the current Propulsion Test Directorate 
electronic work control system for Test Operations. The PTD Work Control System includes work 
authorization and technical instruction documents, such as test preparation sheets, discrepancy reports, , test 
requests, pre-test briefing reports, and other test operations supporting tools. 

The environment that existed in the E-Complex test areas in the late 1990’s was one of enormous growth which 
brought people of diverse backgrounds together for the sole purpose of testing propulsion hardware. The problem 
that faced us was that these newly formed teams did not have a consistent and clearly understood method for 
writing, performing or verifying work. A paper system was developed that would allow the teams to use the same 
forms, but this still presented problems in the large amount of errors occurring, such as lost paperwork and 
inconsistent implementation. In a sampling of errors in August 1999, the paper work control system encountered 
250 errors out of 230 documents released and completed, for an error rate of 111%. Errors in technical instruction 
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documents such as the ones used in Test Operations can result in death, injury, loss of capabilities, or not meeting 
the project’s objectives. This was unacceptable and our group responded quickly to rectify the problem. 

The system developed needed to help the engineers and technicians communicate and prevent repeating past 
mistakes as well as prevent errors before releasing to work. By capturing critical data and tracking the progression 
of work, the proposed new PTD Work Control System would increase the work efficiency of engineers and 
technicians that support design, construction, activation, and testing of propulsion test projects at Stennis Space 
Center. 

During the development of the basic system, additional goals were introduced; improving communications and 
collaboration between the many end users. I initially developed the E-Complex Work Control system in 1999 and 
released it for production in October 2000. Today’s PTD Work Control system evolved by incorporating 
improvements and automations suggested by field technicians, test engineers, designers, safety engineers, and 
project managers. 

The PTD Work Control System development leveraged existing investments in tools and products by expanding 
them into an integrated collaborative engineering environment. The technical problems were many and varied; the 
challenge has been to remain innovated and proactive in building this collaborative environment without spending 
vast sums of money. The path has been incremental and value-oriented. More importantly, it has increased the 
quality of work provided to our customers without increasing the cost of doing business with PTD at Stennis Space 
Center. 


II. Design and Structure of PTD Work Control System 

The foundation of the PTD Work Control System was built using an off the shelf software called FileMaker 
Pro 1 . In 1999 the software had already been used in the Test Operations areas of NASA and Boeing Space Shuttle 
Main Engine Testing. It was proven to be reliable and very easy to customize to our specific requirements. In 
today’s lean testing environment it is imperative to design functional systems or also called data products as to 
decrease short term and long term reoccurring cost, i.e. overhead that is passed on to the customer. This tool met 
this need. 

Utilizing the graphical interface tools provided within the software each screen was designed to meet the needs 
of various users such as: test engineers, designers, technicians, supervisors, auditors, configuration management 
personnel, and other support contractors. An example of an interface screen can be seen in Figure 1. When the 
basic interface screens were designed, automation of critical tasks and/or controls was programmed using the 
scripting language within FileMaker Pro 1 , Figure 2. These two elements combined are known as a data product, 
which is a well defined electronic process that serves as a tool to accomplish a predetermined task. All of the data 
products used by Propulsion Test Directorate (PTD) to accomplish daily test operations are combined into one 
system and accessed through common screen and is known as the PTD Work Control Screen. 



Figurel: Example of User Screen 



Figure 2: Example of Programming Script 


The ease of programming and interfaces’ customization found in the software’s foundation led to a development 
process known as rapid prototyping. This allowed a typical data product to be created in the time frame of one week. 
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The most complicated data product designed and implemented took only two months to complete including user 
testing. The screens were designed to be simple and non-cluttered because this system is used daily and I found less 
human error occurred with the simpler interfaces. 

Not only is does this system allow for easy development, it allows real time modifications or improvements to be 
made without taking the system offline. This feature is critical to test operations since they rely on the system daily 
to perform and track tasks that support mission goals. The only items that currently can not be changed real time are 
passwords and field definitions, because the fields are shared to multi-users (users are required to temporarily log 
out). Another useful feature of the parent software is that each data field is automatically saved after each entry. 
The structure of the software allows simple searches on any field or for more complex multi-criteria searches. 
Essentially the user entry screen can become an adhoc search screen in addition to the specialized search screens, 
The users often prefer defining their own criteria over other search techniques. 

The total initial investment to create the system and provide it to all of PTD, 150 personnel, (using about 60 
workstations) was approximately $6000 in 2000 and one-half of a full time employee (FTE). The only additional 
cost that has been incurred since the initial investment has been in consulting services to build additional screens and 
implement improvements at an approximate cost of $55K over three years. In early 2004 the production version of 
FileMaker Pro was upgraded from 5.1 to 6.0 that were included in a maintenance agreement of approximately 
$2500. 


III. General Description of PTD Work Control System Components 

There are many components of the PTD Work Control System (WCS). They support the various functions used 
in preparing for testing from planning work to recording discrepancies. The system has twelve primary data 
products or user interfaces, split into three groups, one per test stand, El, E2, & E3. 

Figure 3 shows the Main Screen for the system. The 
goal of the main screen design was to keep it simple and 
easy to identify what and where the data products are. 

Along with the primary data products, there are nine 
supporting data products. One critical support data 
product is the Test Open Items report, which summarizes 
all open items for a specific test stand and/or project that 
includes open electrical and mechanical TPSs and any 
open Discrepancies. This interface also offers four links 
to other PTD/Stennis systems They include an enterprise 
system that automates the configuration management of 
PTD’s drawings/designs, a component portal that 
searches into four different component databases on site, 
a test data web site, and the PTD home page of S SC’s 
Intranet. 

Each button on the screens represents a script that 
controls user flow and improves quality. The WCS has 
over 300 scripts that perform various functions. Below 
the major PTD Work Control System data products’ 
function and unique capabilities are described. 

A. TPS - Test Preparation Sheets 

The TPS is the heart of test operations. It is the vehicle by which technical tasks are communicated to those who 
will perform the tasks. It also serves as an approval document to do the work or tasks, safety and hazards 
identifiers, quality inspection requirements, and engineering verification that the tasks were completed to 
specifications. 

The creation phase of the TPS offers the engineers an option to search for like activities within the system and 
duplicate content if needed. It also has built in help features that insure that the engineer enters hazards and that the 
appropriate personal protection equipment, PPE, is called out within the TPS. Other features include: 1. verification 
that each field is filled in with appropriate data, automated time and date stamps, 2. Verification that the TPS is 
ready to work, 3. Locking feature when the TPS is closed by the Work Control Coordinator, and many more. Figure 
4 shows the entry screen for a TPS. 



Figure 3. Main Screen for PTD 
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Upon completion by the engineer, the TPS data product has a built in workflow that allows the engineer or 
supervisor to know the status of the work. It provides multiple reports as seen in an Open TPS List with ability to 
set priority by the Test Director of the specified test facility. 

Once the engineer is done writing the TPS, it is printed out and he or she obtains the required signatures. Then it 
is dropped in a box for the Test Operations Contractor’s facilitators. The facilitator has their own user screen which 
helps them track status, parts, and cause of delays in working the TPS. An example of the Facilitator’s screen can 
be seen in Attachment A. When developing this system, I with my management made the decision to keep the part 
that actually performs the work as a paper based system. This was due to the remoteness of the many test facilities 
from workstations and the complex safety issue of wireless computers in hydrogen environments. There was also an 
advantage to introducing automated data products in small phases; this allowed the work force to increase their 
comfort with using a semi-automated system gradually. The users responded with many suggestions to improve the 
system and therefore making it “their” system. 

B. Discrepancy Reports 

A Discrepancy Report is written when a discrepancy occurs which is defined as an anomaly or failure of a 
component, data, or system that requires repair, replacement, or explanation. Anyone is allowed to write a DR, but it 
must be reviewed by the facility’s Test Director. He or she has technical responsibility for the test facility. 

The PTD Work Control System has a title page where the discrepancy is described in detail. There is also an 
option to email the other TD’s and the Office of Safety and Mission Assurance if the writer believes this discrepancy 
might impact others. This is known in the system as a Corrective Action Request. There is a recently added feature 
that allows the user to also create a “lesson learned” in the PTD Lesson Learned system that resides in Stennis’s 
Design and Data Management System, (DDMS), from the main DR title page. DDMS is the system that controls 
configuration management of Facility and Project drawings, documents, and has a PTD Lesson Learned data 
product. DDMS was developed using Windchill Foundation 2 software and customizing it to Stennis’s needs. 

A Test Operations Engineer, TOE, is required to answer or research any logged discrepancy. They accomplish 
this by completing a DR Disposition. If the solution is not known they would number the dispositions as Partial 
Dispositions numbered 1,2,3, etc. When a solution is determined, the TOE writes a ’’Final Disposition” and closes 
the DR. The DR Disposition data product looks and functions the same as the TPS data product. It is also a work 
authorization document. The DR title page and subsequence dispositions all have the same number but can have 
different authors. This makes locating a discrepancy easier and has helped in identifying reoccurring problems that 
led to a resign of a critical component. 



C. Test Requests 
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A Test Request, TR. is written for each test by the assigned System Integration Engineer with customer input. 
This is the authorization to perform a specific test. The TR also details needed to set up and actually run a specific 
test are called out in the TR. The PTD WCS provides automatic numbering of the TRs and a generation screen for 
Flash Reports. This is a quick post test summary of the test results that is sent to all involved parties and their 
management. The Test Request typically has multiple attached Detailed Operating Procedures, DOP’s that do 
specific tasks. These together with a pretest Open Item report, make what is known as a Test Package. The Test 
Packages are audited using the same method and interface as a TPS or DR. A typical Test Package has 
approximately seven Detailed Operating Procedures. The DOPs reside in the Stennis DDMS system and are under 
configuration management. 


D. Installed Component Configuration Database 

Another important component of testing is verifying that the components used to obtain test data meet specific 
design parameters and clean levels. This is most critical in the liquid oxygen, liquid hydrogen, and hydrogen 
peroxide systems. A data product was designed in the PTD Work Control System where component certification 
information from a TPS or DR is entered. 


This data product is known simply as the Component 
Configuration Database. Although it is much more than 
a simple database, it has multiple input screens with pull 
down menus and re-certification reports. This system 
was only intended to keep currently installed component 
information. Out of necessity, the data product has been 
customized to keep previously installed components’ 
information as well, called historical data. The data is 
beneficial to planning and designing for new projects or 
returning projects. An example of the user interface 
screen used by test operations, refer to Appendix A. A 
copy of the test stands Component Configuration 
Database can be retained for each test to show test stand 
configuration. Examples of the type of component 
information that is retained is locator number, 
manufacture, model number, serial number, clean level, 
service or medium intended for, drawing number, 
working pressure, and temperature range. There are 
approximately 52 different data field available for 
storing component information 



Figure 6. Component Configuration Database 


E. Other Misc. Support Features 

The PTD Work Control System also supports the System Integration Engineers, (SIE), group by making all of 
their tools and forms readily available. They use the PTD WCS to track, verify, and validate requirements. One of 
the most used tool is called a Change Request, CR, which is required to change a pre-agreed to requirement. This 
CR has the schedule and budget impact information as well as the signatures required to accept the change. Other 
SIE tools are Data Release forms, work flow diagrams for SIE processes, and schedule impact database. Examples 
of these can be seen in Attachment A. Many of the initial designs for the SIE tools were done by the SIE group 
leads, Ms. Christine Powell and Mr. Brad Messer. 

Another feature of the PTD WSC is the Pre Test Briefing, which is a summary of all open TPS, DRs, and TRs 
for a specific test facility and project. When a Pre Test Briefing report is run for a specific project, all open work 
against the facility is shown with the project’s open work. It is critical to see the facility work, because the project is 
so intertwined with the facility systems. Recently it was demonstrated that even the fail safe systems of the Control 
Building’s fire alarm system can affect the project’s testing by cutting the Several links to other systems used by 
TOEs are found on the main PTD WCS screen such as Component Portal, DDMS, Test Data, and PTD Home Page. 

The Component Portal is a Search engine that looks into all of the various component databases. This is a useful 
tool to help locate stored components and also to find component data discrepancies. Stennis ’s DDMS, Design and 
Data Management System, is as a customized system built on Windchill Foundation and manages the configuration 
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of Drawings, Project Requirement documents, PTD Lesson Learned, and Detailed Operating Procedures. DDMS 
PTD drawings are linked to the component data housed in the PTD WCS’s Component Configuration Database. 
Test Data is a web link to a secure server where all of the test data is stored and is password protected. 

Also located on the main screen for the PTD WCS are best practices for writing, and verifying TPSs, DRs, and 
DOPs. They have been developed over 10 years of testing at Stennis and lessons learned from other test areas such 
as Marshall Space Flight Center’s. A link to Stennis’ s official Work Instruction system, called Tech Doc, is 
provided for the engineers to access the latest Stennis Operating Instructions that are requirements for performing 
work in PTD. 


IV. Measured Results of PTD WC System 

After the initial system was in production for a couple of months, it occurred to me that we needed a way to gage 
if it was indeed decreasing our error rate in performing work. Two methods are used to gage effectiveness of the 
system, individual audits of each work authorization document and trending the data long term. The trending 
program was developed primarily by Mr. William (Bud) Nail of Technological Services, Inc. under a NASA 
contract. 

A. Tracking of Tasks Work Errors 

The audit of work authorization documents, i.e. TPS’s, DR’s and Test Packages, is performed by the group’s 
Work Control Coordinator. She audits each document as it is turned in for closure, in paper form. She has 
approximately thirty quality items she looks for, that are best practices or requirements per the Stennis Operating 
Instruction, SOI-80 80-0027 3 . Attachment B shows the entry screen for capturing error data and the types of 
problems that are checked for. If a new problem occurs, it will be added to the list for the next audit. 


B. Long Term Results 

Once individual data is collected, I run a program to collect all problems from the multiple test facilities and 
create monthly reports. Figure 7 shown below is a sample of a monthly report and Figure 8 is the trending report 
that is updated monthly as well. 



John C. Stennis Space Center 


E-Complex Work Contol Issues vs. T otal WADs Closed 
Aug 2000 - Aug 2004 


Work 

;‘<=' 


— ^ Electronic Work 
Control System 


■ Total Closed 

■ Tot TPS/D FVTR Issues 


Figure 7. Sample Monthly Errors Report for WCS Figure 8. Four year Trend Report on WCS Errors 
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C. Limitations of System 

The major limitation of the current WCS is that signatures are still obtained manually. During initial 
development this was the users’ desire. Now that the users have become more computer savvy, they would 
welcome automated signatures. The engineers now desire automated signature capability because they are no longer 
all in the same vicinity and obtaining signature in a timely manner has become difficult. The other limitation is that 
the system does not have a built in work flow with notifications of completed dates missed. This could be 
programmed into the system, but a study on the return on investment needs to be completed first. 


D. Ongoing and Planned Modifications 

Currently the Test Area support groups such as Maintenance, Cryo Facility, Water Plant, and High Pressure Gas 
Facility are being moved from an all paper based system to the PTD Work Control System. Their requirements 
have been gathered and the beta version is expected to be completed by mid January 2005. Plans are being 
developed to link the PTD Work Control System & DDMS PTD Lessons Learned system with Active Risk 
Management, ARM 4 . 

Also in work is the integration of TPSs, DRs, & TR’s with project schedules. This is being pursued to 
implement Earned Value Management to a lower level than is currently available. This would allow Test 
Operations to better report value of work performed and to respond quickly when corrections are needed to meet end 
project goals. 


V. Conclusion 

The PTD Work Control system is working today in great part to the high degree of user input to the design and 
modifications. This system continues to be the backbone of how work is done within the E-Complex Test Area. It’s 
most useful feature, for supporting an R&D type of business like that of Stennis’s, is its ease of customization and 
dependability. This PTD WCS system is only a tool and has not become a legacy system that requires more money 
and support to maintain it, than the actual cost of the work it helps conduct. Even though capturing knowledge and 
making it accessible were the end goals of this system, the system itself was not planned to be the only long term 
system ever needed; all data within the system can be easily migrated to future systems if needed. 

The contributions of the PTD Work Control System are hard to quantitatively measure; one method might be to 
measure lost productivity if the system were not there. In the world of Propulsion Testing cost is important, but for 
most of our customers, schedule seems to be the driving force for the projects. This is where user tools or systems 
that are responsive and easily modified can make the most impact. I would not say that the PTD WCS system is 
complete, because as we grow and leam so will the work control system. 
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Appendix A 

Samples of user interface screens and reports. 
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Appendix B 

1. Work Control Coordinator’s Entry Screen for performing audits. 


WCView ▼! 



Records: 

3606 


Found: 
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Unsorted 


Work Control Report TPS Information 
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2. Work Control Coordinator’s List of problems to verify in performing an audit. 


1- Not Identifying as Safety Critical per SPG 
8715.1, SSC Safety Manual. 

2- Not Identifying Drawing No. 

3- Not providing a Need Date 

4- Not listing Potential Hazards 

5- Not listing Test Request No. on TPS 

6- Not referencing DR No. on TPS when a 
problem occurred. 

7- Not assigning TR No. to attachments. 

8- Turned in Test Pkg w/o going thru System 
Integration Engineer or TD 

9- Using Pencil on TPS/DR/DOP 

10- Not Referencing EO for Conf. Change 

11- Pre-Op Briefing not Signed/Dated 

12- Worked Unsigned/Unapproved DOP 

13- No Parts List 

14- Not using red ink for changes 

15- Not using Mandatory Inspection Points, 
MIPs, when required 


16- Not including ”TPS Complete" Step 

17- Not attaching Completed Configuration 
Control Doc. 

18- No Sign./Verification of Closure or 
Completeness 

19- Not enough detail in instructions 

20- No Estimated Man-Hours 

21- Lost Original After Work Complete 

22- No DR Discrepancy Page 

23- No Peer Review Signature 

24- No Safety Review/Signature 

25- No Schedule Reference or Constraint 

26- No TD Approval on DR 

27- Not Buying all required Steps 

28- Not Filling in Blanks in Body of WAD 

29- Not initialing & dating changes/redlines 

30- Not Ref. TPS on Attachments 
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